Search for: All records

Abstract Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the unavailability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), typically benefit from the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of Transformers and long short-term memory networks, which constitute model-free RL solutions and work directly on the observation space, with an approach termed the belief-input method, which works on the belief space by exploiting the learned POMDP model for belief inference. We apply these methods to the real-world problem of optimal maintenance planning for railway assets and compare the results with the current real-life policy. We show that the RL policy learned by the belief-input method is able to outperform the real-life policy by yielding significantly reduced life-cycle costs. 

Abstract This work presents a hybrid shear‐flexible beam‐element, capable of capturing arbitrarily large inelastic displacements and rotations of planar frame structures with just one element per member. Following Reissner's geometrically exact theory, the finite element problem is herein formulated within nonlinear programming principles, where the total potential energy is treated as the objective function and the exact strain‐displacement relations are imposed as kinematic constraints. The approximation of integral expressions is conducted by an appropriate quadrature, and by introducing Lagrange multipliers, the Lagrangian of the minimization program is formed and solutions are sought based on the satisfaction of necessary optimality conditions. In addition to displacement degrees of freedom at the two element edge nodes, strain measures of the centroid act as unknown variables at the quadrature points, while only the curvature field is interpolated, to enforce compatibility throughout the element. Inelastic calculations are carried out by numerical integration of the material stress‐strain law at the cross‐section level. The locking‐free behavior of the element is presented and discussed, and its overall performance is demonstrated on a set of well‐known numerical examples. Results are compared with analytical solutions, where available, and outcomes based on flexibility‐based beam elements and quadrilateral elements, verifying the efficiency of the formulation.